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ABSTRACT 

This  paper  describes  the  progress  on  demonstrating  a  methodology  for 
reliability  assessment  for  structural  damage  sensing  (SDS)  systems.  In  particular,  it 
presents  the  initial  results  obtained  from  the  application  of  a  protocol  designed  for 
utilizing  empirical  data,  models,  simulations,  and  uncertainty  analyses  for 
statistically  characterizing  SDS  reliability  for  a  damage  detection  case.  The 
manufactured  test  article  representing  an  aircraft  structure  of  medium  complexity 
consists  of  three  plates  connected  by  two  lap  joints  with  fasteners.  Fatigue  crack 
damage  around  the  fastener  holes  was  simulated  by  manually  created  thin  cuts  at 
selected  locations.  The  test  fixture  design  provides  the  capability  to  vary  critical 
parameters  of  the  system  with  a  focus  on  force  loading  boundary  conditions,  joint 
fastener  torque  conditions,  and  temperature.  The  initial  demonstration  on  this  test 
article  and  fixture  uses  a  low  frequency  vibration  based  damage  detection  method. 
Frequency  domain  metrics  were  utilized  for  studying  changes  in  structural 
dynamics  due  to  mechanical  loading,  thermal  loading,  and  actual  damage.  For  the 
demonstration,  key  factors  that  affect  the  capability  to  sense  the  effects  of  damage 
were  assessed  through  controlled  studies  of  (a)  loading  and  unloading,  (b)  fastener 
torque,  (c)  boundary  condition  variation,  (d)  temperature  variation  and  temperature 
gradients,  (e)  sensor  bond  quality  and  operation  performance,  (f)  ambient  noise,  and 
(g)  sensitivity  to  flaw  growth.  The  design  of  the  full  validation  study  and  the  current 
results  of  the  implementation  are  presented,  highlighting  general  protocol  feasibility 
while  identifying  remaining  challenges  for  full  demonstration  and  broad  use  of  the 
methodology  and  protocol. 


INTRODUCTION 

The  successful  deployment  of  systems  for  health  monitoring  of  structures 
clearly  depends  on  appropriate  verification  and  validation  (V&V)  of  these  structural 
health  monitoring  (SHM)  systems.  The  V&V  method  must  explicitly  evaluate  all 
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aspects  of  the  SHM  system  that  can  affect  its  capability  to  detect,  localize,  or 
characterize  damage.  Moreover,  it  must  evaluate  the  effects  that  usage  and 
environmental  conditions  have  on  these  capabilities  over  time.  The  current  U.S.  Air 
Force  practice  for  maintaining  aircraft  structures  follows  the  Aircraft  Structural 
Integrity  Program  methods,  as  documented  in  MIL-STD-1530C  [1].  A  very 
important  component  of  the  damage  tolerance  approach  and  the  calculation  of  risk 
that  are  intrinsic  to  the  ASIP  methodology  is  the  assessment  of  the  reliability  of 
Nondestructive  Evaluation  (NDE)  methods  that  are  used  for  periodic  inspection  of 
structures.  MIL-HDBK-1823A  provides  guidance  on  probabilistic  methods  for 
NDE  reliability  assessment  and  introduces  the  possible  use  of  models  to 
complement  experimentation  for  the  necessary  probability  of  detection  (POD) 
determination  [2],  As  SHM  methods  depending  on  permanent,  on-board  mounted 
damage  sensing  systems  continue  to  be  proposed  and  developed  for  complementing 
ground-based  NDE  inspections  for  aircraft  structural  integrity  purposes,  it  is 
necessary  that  the  reliability  of  these  damage  sensing  systems  be  assessed  with  a 
rigor  that  is  suitable  and  sufficient  for  the  function  that  they  are  expected  to  perform 
within  the  ASIP  methodology,  be  it  damage  detection  alone  or  damage  detection, 
localization,  and  characterization  [3],  For  damage  detection,  this  necessarily  results 
in  the  need  for  a  POD  determination.  For  localization  and  characterization,  the 
metrics  for  SHM  reliability  and  their  evaluation  process  are  the  subject  of  current 
research  and  development  [4], 

This  presentation  will  describe  the  progress  in  demonstrating  a  methodology  for 
reliability  assessment  for  SDS  systems.  In  particular,  it  will  show  the  results 
obtained  from  the  application  of  a  protocol  designed  for  utilizing  empirical  data, 
models,  simulations,  and  uncertainty  analyses  for  statistically  characterizing  SDS 
reliability  for  a  damage  detection  case  [5], 


EXPERIMENTAL  STUDY  SETUP 

The  system  used  for  this  initial  demonstration  of  the  protocol  is  a  system  for 
detecting  the  presence  of  damage  using  permanently  mounted  transducers.  A  test 
article  representing  an  aircraft  structure  of  medium  complexity  was  designed  and 
built  (see  Figure  1).  The  test  article  consists  of  three  plates  connected  by  two  lap 
joints  with  fasteners.  In  addition,  a  fixture  was  built  for  supporting  the  test  article. 
Fatigue  crack  damage  around  the  fastener  holes  can  be  simulated  by  manually 
created  thin  cuts  at  selected  locations.  The  final  version  of  the  test  article  and 
fixture,  shown  in  Figure  1(c),  differ  from  those  in  the  design  shown  in  Figure  1(a) 
in  that  (1)  the  two  bottom  support  beams  were  manufactured  out  of  steel  instead  of 
concrete  to  reduce  weight  and  facilitate  transportation  and  (2)  the  line  loading 
attachment  was  manufactured  but  not  used,  and  therefore  it  is  not  shown  in  Figure 
1(c).  The  test  fixture  design  provides  the  capability  to  vary  critical  parameters  of  the 
system  with  a  focus  on  force  loading  boundary  conditions,  joint  fastener  torque 
conditions,  and  temperature.  To  study  the  effect  of  mechanical  loads  on  the 
performance  of  the  onboard  damage  sensing  system,  the  assembly  includes  a 
capability  for  changing  the  configuration  of  the  article  support  to  the  fixture  so  that 
selectable  loads  can  be  applied  to  the  structure. 

The  initial  demonstration  on  this  test  article  and  fixture  uses  a  vibration  based 
damage  detection  method.  An  ETrema  brand  Terfenol-D  magnetostrictive  actuator 


was  used  for  band-limited  pseudo-random  excitation  up  to  1200  Hz,  and  the 
dynamic  response  of  the  plate  was  recorded  using  eight  50mV/g  single-axis 
accelerometers  set  to  measure  out  of  plane  motion  and  provide  input  to  change 
detection  algorithms.  The  accelerometers  were  placed  at  locations  that  attempt  to 
maximize  the  ability  to  detect  change  in  the  modal  dynamics  of  the  structure  even 
in  the  presence  of  modeling  errors.  M-Bond  200  adhesive  was  used  for  semi¬ 
permanently  affixing  the  accelerometers  to  the  bottom  of  the  assembly.  Four 
bonded  foil  350-ohm  strain  gages  were  also  installed  on  the  test  article  for  tracking 
the  state  of  stress  under  mechanical  loading.  Finally,  thermocouples  were  also  used 
for  studying  the  effects  of  heating  due  to  excitation  of  the  damage  detection  system 
and  the  effects  of  the  operational  temperature.  The  layout  of  the  sensors  is  presented 
in  Figure  2.  A  LabVIEW  data  acquisition  system  was  used  for  acquiring  the 
required  data.  Variations  in  operational  temperature  were  simulated  by  testing  the 
system  inside  a  carefully  controlled  Thermotron  SE-1200  environmental  chamber. 

The  method  for  inducing  damage  involved  notching  the  area  of  the  ‘skin’  under 
a  fastener.  To  initially  isolate  damage  to  the  skin  alone,  the  test  article  design 
includes  machined  relief  slots  in  the  joining  plate  at  the  fastener  locations,  but  the 
final  induced  cut  stages  went  beyond  these  relief  slots.  A  change  metric  based  on 
the  area  under  the  curve  representing  the  difference  between  two  frequency 
responses  and  an  R-square  metric  were  used  for  assessing  changes  in  structural 
dynamics  due  to  mechanical  and  thermal  loading,  actual  damage,  and  combinations 
thereof,  and  therefore  attempting  to  detect  damage  presence  and  growth.  The 
frequency  band  of  interest  was  from  200  to  1200  Hz. 


Figure  1  Test  fixture  assembly:  (a)  concept  diagram,  (b)  top  view,  (c),  photograph 
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Figure  2.  Location  of  sensors  on  the  test  article 


FACTOR  EVALUATION  STUDIES 


The  initial  experimental  demonstration  followed  the  proposed  protocol.  Before 
designing  the  validation  test  matrix,  the  following  key  factors  were  assessed 
through  controlled  studies:  (a)  mass  loading  and  unloading,  (b)  fastener  torque,  (c) 
boundary  condition  variation,  (d)  temperature  variation  and  temperature  gradients, 
(e)  sensor  bond  quality  and  operation  performance,  (f)  ambient  noise,  and  (g) 
sensitivity  to  flaw  growth. 

Initial  tests  used  a  simple  R-squared  change  metric  to  assess  the  change  in  the 
frequency  response  between  accelerometer  1  and  accelerometers  2-7  as  a  function 
of  mass  loading  and  unloading.  The  structure  was  mass  loaded  in  steps  of  8.6  lbs. 
The  change  metric  was  found  to  be  highly  sensitive  to  this  loading  and,  more 
importantly,  the  change  metric  was  shown  to  be  not  reversible.  For  example, 
completely  unloading  the  structure  did  not  result  in  the  metric  returning  to  zero. 
Since  mass  loading  and  unloading  frequently  occurs  in  aircraft  structures,  the 
reasons  for  this  irreversibility  must  be  understood  before  application  of  dynamic 
response  changes  from  a  baseline  state  to  detect  damage  could  be  used  in  cases 
where  mass  loading  of  this  relative  magnitude  is  expected.  Possible  sources  for  this 
variability  are  varying  contact  conditions  between  the  plates  and  fasteners  and/or 
varying  temperature  conditions  during  the  study. 

A  fastener  torque  study  was  performed  using  a  select  set  of  fasteners  at  torque 
levels  of  75,  100,  50  and  0  in-lbs,  representing  normal  torque,  high  torque,  low 
torque,  and  ‘hand  tight’  conditions.  Changes  from  75  to  50  in-lbs  had  very  little 
effect  on  the  frequency  response  function  (FRF).  Changes  from  75  to  100  in-lbs  had 
more  effect,  but  not  much  more  significant.  Changes  from  the  normal  torque  to  the 
hand  tight  condition  had  a  much  bigger  effect,  as  expected.  Repeated  removal  and 
reinstallation  of  fasteners  will  be  included  in  the  final  validation  study. 

Thermal  loading  studies  were  performed  by  varying  the  ambient  temperature 
from  -20°F  to  150°F.  During  this  study,  the  significant  thermal  capacity  of  the  panel 
end  conditions  fixtures  was  found  to  produce  significant  thermal  gradients  across 
the  test  article.  During  heating  and  cooling  periods,  temperature  gradients  as  high  as 
45°F  across  the  test  specimen  were  observed.  For  validation  studies,  an  estimate  of 
expected  gradients  ‘in  the  field’  is  needed.  An  assumption  will  be  made  for  the 
validation  study  that  temperature  gradients  in  the  region  of  interest  will  be  limited 
to  +/-  10°F. 

Failure  of  accelerometer  bonding  was  observed  several  times  during  thermal 
testing.  These  failures  occurred  during  the  prolonged  high  temperature  runs  at 
150°F.  The  bonds  for  strain  gauges  and  adhesive  tape  for  the  thermocouples  held  up 
well  during  testing.  Reaffixing  the  fallen  and  slightly  loose  accelerometers  greatly 
improved  their  coherence  estimates.  Coherence  is  viewed  as  a  viable  metric  to 
monitor  bond  quality  over  time.  While  vibration-based  damage  detection  systems 
are  proposed  as  global  methods  with  some  sensor  redundancy,  relying  on  a  single 
reference  sensor  will  result,  upon  bond  degradation,  in  either  highly  degraded 
performance  and/or  complete  failure  of  the  damage  detection  system  well  before 
the  structure  end-of-life  is  reached.  Sensor  and  sensor  bonding  reliability  must 
therefore  be  accurately  assessed  as  part  of  a  validation  study. 

The  noise  generated  by  the  environmental  chamber  and  by  other  equipment  in 
the  laboratory  posed  an  opportunity  for  studying  the  effects  of  ambient  noise  on  the 


detection  capability.  Coherence  levels  were  observed  to  change  depending  on 
whether  the  unit  was  cooling  or  heating  and  the  ‘throttle’  level  of  the  chamber,  and 
on  whether  the  chamber  door  was  open  or  closed.  An  excitation  amplification  study 
was  performed  using  coherence  estimates  to  determine  the  optimal  excitation 
amplifier  gain  in  the  various  conditions:  (1)  chamber  on,  door  closed,  (2)  chamber 
off,  door  open,  and  (3)  chamber  off,  door  closed.  The  amplifier  gain  yielding 
optimal  coherence  was  different  for  the  various  conditions,  and  therefore  a 
compromise  level  yielding  high  coherence  for  all  conditions  was  selected  for  all 
subsequent  runs.  For  the  validation  study,  both  ‘chamber  on’  and  ‘chamber  off 
conditions  will  be  acquired  with  the  objective  of  including  in  the  analysis  the 
effects  of  varying  noise  conditions  that  are  expected  in  the  field. 

An  initial  study  on  the  effect  of  damage  growth  was  performed  to  ensure 
adequate  sensitivity  during  the  final  validation  study.  Customized  XActo™  blades 
were  used  to  make  cuts  in  the  aluminum  plates.  The  resulting  notch  width  was 
estimated  to  be  approximately  0.012"  +/-  0.002".  Cuts  were  initially  made  at  0.063" 
(1/16")  increments  up  to  0.25".  For  the  first  series  of  cuts  up  to  0.25",  Figure  3 
presents  a  history  of  the  AREA  damage  metric  during  Damage  Study  1,  for  the 
chamber  off  and  fastener  torque  at  75  in-lb  condition.  During  this  initial  series, 
sensitivity  to  notch  length  increases  was  observed,  but  the  trend  was  small  relative 
to  noise,  and  not  quite  linear.  Greater  sensitivity  to  the  larger  cuts  was  observed  and 
clear  sensitivity  to  notches  on  the  order  of  0.63"  was  demonstrated.  Also,  a 
significant  increase  in  the  damage  metric  was  observed  after  the  two  week  delay 
between  the  end  of  the  0.25"  notch  cut  and  the  start  of  the  0.3 1 "  notch  cut. 
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Figure  3.  AREA  damage  metric,  Damage  Study  1,  200  Hz  -  800  Hz 


EXPERIMENT  DESIGN  FOR  VALIDATION  STUDY 


The  design  of  the  validation  study  only  included  a  single  stage  demonstration 
using  laboratory  testing  of  a  relevant  structure  and  under  expected  environmental 
conditions.  The  validation  study  consisted  of  growing  flaws  by  artificially  cutting 
the  structure  at  two  fastener  site  locations,  site  #2  (fastener  1 1)  and  site  #3  (fastener 
7),  as  shown  in  Figure  4.  A  series  of  environmental  and  joint  boundary  conditions 
were  studied  after  each  flaw  growth  scenario:  temperature  variation  (+/-  40°F), 
temperature  gradients,  loading  and  unloading  of  10  lb  mass,  a  simulated 
maintenance  action  at  a  set  of  fasteners  (see  Figure  4)  including  the  case  of  minor 
loosening,  and  reinstallation  and  replacement  of  accelerometers.  During  the  flaw 
growth  period,  a  series  of  tests  were  performed  including  testing  before  and  after 
flaw  growth,  and  after  fastener  installation.  Temperature  chamber  states  of  ‘on’  and 
‘off  with  the  ambient  temperature  set  to  T  =  72°F  were  acquired  for  all  test 
conditions.  Five  averages  were  taken  for  all  SDS  system  data  acquisitions.  After 
any  environmental  condition  change  or  change  to  the  test  fixture,  testing  was  also 
performed  with  the  ambient  temperature  returned  to  T  =  72°F.  At  the  time  of  this 
writing,  the  results  of  these  experimental  measurements  are  being  analyzed. 
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Figure  4.  Locations  for  flaw  growth  and  simulated  maintenance  actions 


PROBABILITY  OF  DETECTION  ANALYSIS 

A  summary  description  of  the  approach  being  followed  for  obtaining  a 
probability  of  detection  model  for  on-board  damage  detection  cases  follows. 
Conventional  probability  of  detection  (POD)  evaluation  for  many  quantitative  NDE 
applications  first  uses  empirical  data  to  evaluate  statistical  relationships  between  the 
measurement  response, a,  and  the  primary  flaw  size  variable,  a.  Through 
application  of  a  detection  criterion  as  part  of  the  NDE  procedure,  this  statistical  ‘  a 
versus  a  ’  model  can  be  used  for  evaluating  the  POD  curve  and  probability  of  false 
call  (POFC)  rate,  which  together  are  usually  referred  to  as  “a  POD  model”.  The 
detection  system  can  be  abstractly  represented  by  a  set  of  random  variables  ai  that 


act  as  inputs  to  a  measurement  model.  Input  variables  can  be  categorized  as  being 
controlled  (e.g.  flaw  size  and  material  properties)  or  uncontrolled  (e.g.  liftoff,  flaw 
morphology,  and  measurement  noise).  Detection  consists  of  the  measurement 
model  output  a  being  classified  (or  “called”)  according  to  pre-specified  rules  (e.g. 
a  threshold).  There  are  a  number  of  mathematical  formulations  that  can  be  utilized 
for  statistically  characterizing  how  well  this  system  detects  values  of  the  input 
variables  of  interest  using  the  particular  classification  rules.  The  model-assisted 
POD  (MAPOD)  approach  proposes  to  replace  a  conventional  statistical  fit  in  the 
measurement  model  with  a  complete  physics-based  model,  /,  calibrated  for  a  given 
set  of  experimental  conditions.  This  relationship  is  given  by 

<2  =  fio  +  fiif{ai)+£,  (1) 

where  flo  and  J3\  represent  the  model  calibration  parameters,  and  s  represents  the 
residual  error  between  the  model  and  the  experimental  data.  Estimating  the  statistics 
of  /3q,  J3\ ,  and  s  necessitates  specific  experimental  sampling  requirements. 
Variations  due  to  flaw  size  and  environmental  (noise)  conditions,  for  example,  are 
represented  in  the  model  as  probability  distributions  of  the  input  variables. 
Surrogate  models  derived  from  numerical  simulations  can  be  practically 
implemented  to  facilitate  repeated  calls  to  the  model  f.  Hybrid  models 
incorporating  both  empirical  and  physics-based  components  can  also  be 
implemented  to  address  all  key  factors  including  those  that  cannot  be  adequately 
simulated. 

In  this  study,  the  primary  variable  associated  with  the  critical  flaw  size  is  crack 
(notch)  length,  a\.  Controlled  secondary  variables  in  the  study  include  mean 
temperature  (ai),  temperature  gradients  (<23),  ambient  noise  level  (<24),  structure 
loading  and  unloaded  (<25)  and  minor  boundary  condition  variation  /  fastener 
loosening  (ad).  A  response  surface  methodology  can  be  applied  here  to  estimate  the 
effect  of  each  secondary  factor  on  the  damage  metric  response.  Random  events 
such  as  sensor  failure  (or  complete  sensor  disbond)  (hi),  sensor  bond  degradation 
(£2),  sensor  replacement  (hi)  and  local  maintenance  actions  (Zm)  can  also  be 
considered  in  the  POD  evaluation.  Assumptions  concerning  their  frequency  (with 
uncertainty  bounds)  can  be  made  and  empirical  models  representing  their  effect  can 
be  evaluated  and  applied  in  conjunction  with  the  proposed  scope  and  details  of  the 
SDS  application  (inspection  frequency,  SHM  service  cycles,  etc.) 

To  complete  the  POD  evaluation,  an  assessment  of  the  detection  model  under 
varying  input  conditions  including  uncertainty  propagation  is  necessary.  In  prior 
work  [6],  a  second-order  probabilistic  approach  has  been  developed  to  propagate 
both  aleatory  uncertainty,  due  to  inherent  randomness  in  system  behavior,  and 
epistemic  uncertainty,  due  to  a  lack  of  knowledge  about  values  expected  to  be 
fixed.  Using  this  approach,  epistemic  variables  are  specified  as  intervals  on  values 
of  parameters  such  as  the  means  and  standard  deviations  of  random  variables. 
Monte  Carlo  analysis  is  thus  performed  using  outer  and  inner  loops.  The  outer  loop 
varies  the  values  of  distribution  parameters  of  selected  epistemic  variables  while  the 
inner  loop  samples  from  the  distributions.  A  MAPOD  case  study  incorporating 
second-order  probability  analysis  using  Monte  Carlo  simulations  has  recently  been 
demonstrated  [7]. 


CLOSING  REMARKS 


This  work  has  presented  initial  results  of  a  demonstration  featuring  the 
application  of  the  proposed  validation  protocol  to  a  vibration-based  structural 
damage  sensing  system.  Protocol  steps  emphasize  the  importance  of  assessing  the 
key  application  characteristics  and  evaluating  the  significant  factors  that  control 
performance  and  reliability.  Results  of  the  initial  steps  of  the  validation  protocol 
which  evaluate  the  various  factors  that  affect  system  capability  are  presented. 
Finally,  a  brief  description  of  the  statistical  analysis  required  for  processing  the 
experimental  data  and  combining  it  with  models  to  yield  a  probability  of  detection 
model  is  included.  The  paper  highlights  general  validation  protocol  feasibility  while 
identifying  remaining  challenges  for  completion  of  this  validation  study  in 
particular  and  for  broad  use  of  the  validation  protocol  in  general.  Completion  of  this 
demonstration  study,  including  a  POD  model,  is  expected  in  late  summer  2011. 
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